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Abstract 


Game-based learning (GBL) is a popular approach to designing engaging educational experiences 
enhancing motivation and enjoyment while at the same time supporting the attainment of learning 
outcomes. Robot-assisted language learning (RALL) based on game-based learning with humanoid 
robots is becoming an interesting area of human-robot interaction (HRI) which has the potential to enrich 
educational curricula. Studies have shown that interacting with a humanoid robot as part of language 
learning may be beneficial for learner motivation and improvement across a range of language skills, 
especially speaking accuracy, fluency, and pronunciation. Both RALL and GBL can reduce learning 
anxiety and enhance positive attitudes towards learning grammar, which tends to be perceived as boring 
and complicated by language learners. This paper presents an instructional design and programming of 
a game-based learning scenario for learning English grammar in higher education, in which the 
humanoid robot Pepper is used as a game partner. The scenario is called “Make or Do Game with 
Pepper” and is intended as an educational, interactive quiz in two versions, i.e. a single player version 
and a team-based version. This paper presents the first version of the game for one-on-one play 
between the human and the robot. This version of the game was tested with a small group of five 
students in Digital Business Studies. The game consists of two main parts: the interaction with the 
Pepper robot and the tablet application which is fixed to the chest of the robot. The results from the pilot 
study show that students enjoyed playing the grammar game with the Pepper robot, and perceived the 
robot as a friendly, kind, pleasant, funny and relaxed game partner which appeared good-humoured, 
jocular, funny, relaxed and interested. The paper also describes what the participants liked and what 
they did not like about the game, and which recommendations were made for future iterations The 
paper ends with a conclusion and recommendations for further research. 
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1 INTRODUCTION 


Game based learning (GBL) is a popular approach to designing engaging educational experiences 
enhancing motivation and enjoyment while at the same time supporting the attainment of learning 
outcomes. A review of 114 published papers on game-based language learning, reveals that game- 
based approaches to language learning focus primarily on vocabulary acquisition, specific input skills 
(mostly reading and listening) and output skills (mostly writing and speaking) [1]. The most frequently 
occurring elements in game-based language learning designs also include different feedback types 
(such as corrective feedback and scores for correct responses), themes (including settings or characters 
consistent within a game), narratives (including different techniques of storytelling), points (such as 
points for completing tasks, challenges, battles), as well as levels and progression (mostly related to 
increasing difficulty and improvement of skills) [1]. Other elements include such mechanisms as time 
pressure, exploration and signposting [1]. At the same time, there are some gaps in research, for 
example, research on game-based learning of grammar language skills have been scarce [1]. Moreover, 
most of the published papers have focused on children and only a few studies were conducted with 
adult learners and university students [1]. From this perspective, this paper adds to current research by 
providing results from a pilot study with a robot-assisted game-based application for the learning of 
grammar skills for university students. 


Robot-assisted language learning (RALL) is becoming an interesting area in human-robot interaction 
(HRI) and has a potential to enrich educational curricula. Studies in the area of RALL have explored the 
uses of a wide range of educational robots programmed to serve as teaching assistants. A number of 
studies have emphasised specific benefits for language learning, such as creating a relaxed learning 
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experience with less social pressure, enhancing the self-confidence and motivation of language 
learners, and contributing to the diminution of the strategy of many language learners to avoid speaking 
a foreign language they are learning [2]. Educational robots, including humanoid robots, provide a 
tangible embodiment of technology, which has been shown to enhance interaction with and provide 
engaging learning experiences for language learners [2]. A number of recent studies have shown how 
language learners may benefit from using educational robots. For example, the study by [3] indicated 
that using a robot and a tablet to support learning of English of EFL adult learners is beneficial not only 
for motivation but also for the improvement of a range of language skills, especially speaking accuracy, 
fluency, and pronunciation. 


As proposed by [2] humanoid robots have at least ten characteristics that may be exploited to support 
language learning. These are (1) repeatability (repeated practice without fatigue or negative emotion 
enhances comprehension, skill development and familiarity), (2) flexibility (robots can be programmed 
in an individual way and educators do not have to rely on specific instructional materials developed by 
a manufacturer), (3) digitalisation (especially in respect to the flow and documentation of data related to 
learning), (4) humanoid appearance (which makes robots to a more engaging technology which 
enhances curiosity and motivation to learn), (5) body movement and motion (which helps combine 
speaking with gestures), (6) interaction (such as engaging in dialogues and reacting to human 
responses based on language recognition), (7) anthropomorphism (contrary to other educational media, 
humanoid robots are perceived as human-like language speakers), (8) sensing capability, intelligence 
and automatic speech recognition (which enable robots to sense the environment and communicate 
with humans and computers), (9) Janguage understanding and dialogue management (robots can be 
programmed to understand the utterances of language learners despite errors), (10) emotional 
expression (robots can represent different emotions such as joy, disapprobation, sadness, or 
embarrassment, and can make accompanying gestures such as waving or cheering) [2]. 


The use of games in language is not uncommon and a current meta-analysis on the effectiveness of 
using games strategies in English language learning and teaching indicated that games may support 
the development of a range of language skills including vocabulary, reading, writing, listening and 
speaking, but that it is uncommon to use games in teaching grammar skills [4]. Although game-based 
scenarios for Robot Assisted Language Learning (RALL) have been advocated by a number of authors 
[5], design and research in game-based RALL with humanoid robots is still in its infancy. So far, there 
have been only a few publications specifically related to game-based robot assisted language learning 
using humanoid robots. There are a few studies which explore the role of feedback during game-based 
interactions of children with educational robots in the context of language learning. For example, 
using game-based approaches in RALL was shown to enhance the motivation of language learners [5]. 
In the “I spy with my little eye” game, the humanoid robot was effectively employed as a tutor providing 
children with different types of feedback to facilitate second language learning [6]. Another study 
conducted with children showed that playing a storytelling game with a social robot was beneficial for 
language learning, especially for the emulation of the robot's he robot’s use of language, as well as in 
the area of new vocabulary acquisition [7]. In a review of 22 RALL journal articles, [8] Randall reports 
that the most common language teaching methodologies used in RALL are CLT (Communicative 
Language Teaching), TPRS (Teaching Proficiency through Reading and Storytelling), TPR (Total 
Physical Response), and ALM (Audiolingual Method), and that most RALL studies focus on 
schoolchildren aged between 7 and 12 years, as well as pre-schoolers (aged between 4 and 7). The 
authors suggest that the reason for this may be because these are critical and formative years for 
language learning [8]. While most of the studies in RALL in the past were conducted with children, 
studies on game-based RALL with adult populations are now becoming more common. For example, a 
study involving L2 language learners (i.e. second language learners) with the support of an 
anthropomorphic robot, aimed at determining a suitable pedagogical set-up for adult learners, concluded 
that the adult learners were most active when the robot encouraged learner to learner interaction [9]. In 
another study focused on adults, the robot head Furhat was used to test the HRI as well as the human 
to human interaction between test subjects in teaching scenarios where the robot assumed different 
interlocutory roles [10]. The authors concluded that RALL may be used successfully with adult learners, 
but that the proficiency level of learners has a determinative effect on the specific interlocutory role that 
the robot should play [10]. 


2 METHODOLOGY 


The humanoid robot Pepper was applied as the robotic platform for the English language grammar game 
“Make or Do Game with Pepper” for adult university students. Iterative methodology was followed to 
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develop the application for the Pepper robot. The development included a number of phases such as 
requirement engineering, design, implementation and testing. The following sections describe these 
phases in more detail. 


2.1 Requirements engineering 


In the first step of development, a sample application for the Pepper robot was demonstrated in a 
university language learning class to understand whether adult learners (students) were interested in 
game-based learning with a humanoid robot at all. In the next step, the requirements of the “Make or 
Do Game with Pepper” application were collected based on students’ interest during their interaction 
with the Pepper robot in the classroom. The first results showed that students were interested in playing 
the game with the Pepper robot and some of the key functional requirements included: (1) the application 
must have a practice and test option; (2) the robot must understand the voice command Yes/No; (3) the 
robot should produce sad/happy animation depending on student's answer; (4) the application must 
read questions, answers and write student scores to the Firebase database; (5) the application must 
display the score, the descending time limit, and a verbal warning before the time limit expires; (6) 
teachers must be able to change the questions and answers in the database. Besides these functional 
requirements, some further requirements were added based on the past experiences in designing game- 
based applications, such as shuffling questions, not repeating questions during the test, Pepper robot 
loudly asking students questions using speech, and filling in the correct answer with green colour when 
the given answer by the students is wrong. 


2.2 Application design 


The design of the application “Make or Do Game with Pepper’ is based on the iterative design procedure 
and used use case diagrams and architecture diagrams to visualise the design process. The use case 
diagrams were used to summarise the information about users or actors in a system and their 
interactions with the system. Fig. 1 shows the current use case diagram of the “Make or Do Game with 
Pepper” application. 
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Figure 1. Use case diagram of the “Make or Do Game with Pepper” application. 


Two primary actors (student, teacher) and one secondary actor (robot) are used to represent the use 
case diagram of the application. The student can start the game and select an option either Test or 
Practice. Once the student selects the Test or Practice option, the actions such as Sad/Happy, Time 
limit warning, Ask question, are started sequentially. At the end of the practice or test, the student can 
evaluate the application. The teacher can edit the questions, answers and the number of questions 
asked in the test or exam using the database settings. The Pepper robot produces sad and happy 
animations based on the answers from the student. It also produces a welcome animation at the 
beginning and a closing animation at the end of the session. 
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To design the programme logic, the Model-View-Controller (MVC) architecture pattern was used as a 
popular software development pattern. MVC pattern splits the application into 3 components: Model, 
View and Controller. Where the Model component of the “Make or Do Game with Pepper” application is 
responsible for the communication with the database, the View component is the user interface that 
offers interaction to the users and the Controller component establishes the relationship between the 
Model and View component. The Controller component contains the core application logic and activates 
the robot animation. Fig. 2 shows the applied MVC architecture pattern. 
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Figure 2. Architecture of the “Make or Do Game with Pepper” application. 


2.3. Implementation 


The Android Studio IDE (Integrated Development Environment) and the Kotlin programming language 
were used to develop the algorithm of the "Make or Do Game with Pepper" application. Integrating third- 
party libraries for text-to-speech, speech-to-text, and robot animations was the main challenge in 
implementing the “Make or Do Game with Pepper” application and meeting the requirements listed in 
the section above. Fig. 3 shows the flowchart of the algorithm used in the application. 
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Figure 3. Flowchart of the “Make or Do Game with Pepper” application algorithm 
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Once the start button of the application is pressed, the Pepper robot starts the welcome animation and 
speech. After the user's voice input, the application asks for the user name and displays the options, for 
example, Test or Practice. Once the user selects the Test or Practice option, the application displays 
the question and answer from the database, and at the same time, the Pepper robot speaks the question 
aloud with a blank in the position of the answer. For the test option, there is a time limit for the user to 
answer, but for the exercise option, there is no time limit. The Pepper robot produces a sad or happy 
animation depending on the user's answer. At the end of the session, after 10 questions or so, the 
application displays the user's final score and asks for a rating. As soon as the user presses the button 
to exit the application, the robot starts the closing speech and animation, and stores the score, rating, 
and the user's name in the database. 


2.4 Testing of the “Make or Do Game with Pepper” application 


The humanoid robot Pepper is controlled by the Android tablet attached to its chest according to the 
instruction manual provided by the manufacturer SoftBank Robotics. SoftBank Robotics' Kotlin libraries were 
integrated to animate the Pepper robot in order to develop the "Make or Do Game with Pepper" application. 
Several tests were performed to verify the stability of this application and the application passed all the tests 
with a good result. Fig. 4 and Fig. 5 show the testing result and Pepper robot animations. 
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Figure 5. Pepper robot animations of the Make or Do Game application. 


In the first phase of Fig. 4, when the user presses the circled start button, the Pepper robot starts the 
welcome animation shown in Fig. 5 in parallel with a welcoming speech such as "I am very pleased to 
meet you. May | know your name? Please say yes or no." Once the word “Yes” is recognised by the 
Pepper robot, the second phase of Fig. 4 appears. The user must enter the name and press the "Submit" 
button in the second phase to take a test or for the practice. In the 4th phase of Fig. 4, the user presses 
the wrong answer in the test option of the 3 phase, which turns red and the correct answer turns green. 
In this phase, the sad animation of the Pepper robot is activated in parallel with the encouraging speech 
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for the next question. The same happens in case of a correct answer from the user with a happy 
animation. In phase 5 of the process of Fig. 4, the score is displayed along with the user's name and the 
Pepper robot asks for a rating. If the user presses the “Exit” button at this phase, the Pepper robot starts 
the closing animation shown in Fig. 5 in parallel with the closing speech and stores the data in the 
database shown in Phase 6 of Fig. 4. The process continues until the “Make or Do Game with Pepper" 
application is turned off. 


2.5 Design of the study 


The pilot study took place on university campus in June 2022, at the end of the summer semester 2022. 
The participation in the study was voluntary. The pilot study consisted of three parts and started with 
students answering initial questions in an online survey. Next, the students participated in the “Make or 
Do Game with Pepper” and then, in the last part, answered follow-up questions in the same online 
survey. The key research questions for the pilot study were: What are the expectations of students 
towards a humanoid robot facilitating a language learning game? How did students perceive the 
humanoid robot Pepper as a game partner? How did students experience the language learning game 
with Pepper and which improvements to the game would students recommend? The following sections 
present the results from the pilot study including the characteristics of the study sample and the results 
related to each of the research questions. 


3 RESULTS 


The results from the online survey during the pilot study with five students are described below in relation 
to each of the research questions. The IBM SPSS software, version 27 was used for the analysis. 


3.1 Study sample 


The participants in the study were students in the Bachelor study program in Digital Business, who 
participated in the course on Learning Design in the 6th semester of their studies. In total five students 
filled in the online-survey before and after playing the “Make or Do” game. The sample was comprised 
of 60% male and 40% female participants. 60% of the participants were aged 20 to 24 years old, 20% 
30 to 34 years old and 20% 35 to 39 years old. 


3.2 Expectations and perceptions of the Pepper robot 


The first two research questions for the pilot study were: (1) What are the expectations of students 
towards a humanoid robot facilitating a language learning game?, and (2) How do students perceive the 
humanoid robot Pepper as a game partner? There were altogether four questions in the online survey 
related to these research questions. The first question about the expectations, i.e. What do you expect 
from the game “Make or Do” with Pepper?, was asked before the game and the second question related 
to expectations, i.e. How well did the game meet your expectations?, was asked after playing the game. 
The questions about the perception of the Pepper robot as game partner were asked after the game 
and included two questions: How did you perceive Pepper as a game partner?, and How did you 
perceive the mood of Pepper as a game partner? 


The questions were answered by the participants in the following way: 


1 What do you expect from the game “Make or Do” with Pepper? This question was an open 
question in which students could write their own answers. Students expected fun and games, 
human-like interaction with the Pepper robot and thought they would have to imitate something 
Pepper says. 


2 Howwell did the game meet your expectations? This question was rated on a scale from 1 = “my 
expectations were not met at all” to 6 = “my expectations were fully met”. The mean value was M 
= 4.80 (SD = .837) indicating that most students felt that the game met their expectations. 


3 How did you perceive Pepper as game partner? This question was answered by rating five pairs 
of semantic items from the Likeability Scale of the Godspeed questionnaires [11]. Using semantic 
pairs of adjectives is a semantic differential method, which is popular in research on human robot 
interaction. For example, the popular Godspeed questionnaires include five questionnaires using 
semantic differential scales used to assess perceptions of different attributes of robots such as 
anthropomorphism, animacy, liveability, perceived intelligence and safety [11]. Unlike the Likert 
scale, wich presents a series of statements on a symmetric agree-disagree scale, the semantic 
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differential scales ask participants to choose the most appropriate anchor items between two 
bipolar words [11]. Data obtained from semantic differential scales can be treated just as any 
other rating data [11]. The study presented in this paper used one of the five Godspeed 
questionnaires related to “Likeability’. The Likeability Scale consists of five semantic pairs of 
words. The items are rated on a scale from 1 = “unlikeable”, “unfriendly”, “unkind”, “unpleasant”, 
“awful” to 6 =. “likeable”, “friendly”, “kind”, “pleasant”, “nice”. Table 1 below summarises the results 
from the analysis of data related to the Likeability Scale. 


Table 1. Perception of the Pepper robot as a game partner. 


unlikeable (1) | unfriendly (1) | unkind (1) | |unpleasant (1) | awful (1) 

likeable (6) friendly (6) kind (6) pleasant (6) nice (6) 
Mean 5.00 5.40 5.60 5.20 5.60 
Std. Deviation 1.000 .894 .894 1.304 .894 


These results show that high values equal to 5 and above 5 points where reached on the scale 
from 1 to 6, which indicates that the participants perceived the Pepper robot as a likeable, friendly, 
kind, pleasant, and nice game partner. Additionally, the internal consistency of the scale was 
computed and resulted in a high value of a=.934 (Cronbach's Alpha), indicating the excellent 
consistency of the Likeability Scale. 


4 How did you perceive the mood of Pepper as game partner? This question was answered by 
rating five pairs of semantic items on a scale from 1 = “sad”, “serious”, “ina bad mood”, “stressed”, 
“bored” to 6 =. “merry”, “funny”, “in good mood”, “relaxed”, “interested”. These pairs of words were 
inspired by Godspeed questionnaires and self-constructed to assess the perception of the robot's 
mood. Table 2 below summarises the results from the analysis related to the perceived mood of 


the robot. 


Table 2. Perception of the mood of the Pepper robot as a game partner. 


sad (1) serious (1) | in bad mood (1) | stressed (1) bored (1) 
merry (6) funny (6) | |in good mood (6) |_relaxed (6) _| interested (6) 


Mean 5.40 5.20 5.40 5.00 5.20 
Std. Deviation 894 1.304 894 1.000 1.304 


These results show that high values equal to 5 and above 5 points where reached on the scale from 1 
to 6, which indicates that the participants perceived the Pepper robot to be merry, funny, in a good mood, 
relaxed, and interested. Additionally, the internal consistency of the scale was computed and resulted 
in a very high value of a=.986 (Cronbach’s Alpha), indicating an excellent internal consistency of this 
newly developed scale. 


3.3 Experience of the “Make of Do” game with Pepper 


The next research questions for the pilot study were: How did students experience the language learning 
game with Pepper and which improvements to the game would students recommend? There were 
altogether three open questions in the online survey related to these research questions. Participants 
could formulate their own answers to each of the question, i.e. What did you like about the “Make of Do” 
Game with the Pepper robot?; What did you not like about the “Make of Do” Game with the Pepper 
robot?, and Which recommendation do you have for the improvement of the “Make of Do” Game with 
the Pepper robot in the next iterations? 


The questions were answered by the participants in the following way: 


1. What did you like about the “Make of Do” Game with the Pepper robot? Students emphasised the 
following elements which they liked about the game: the interaction with the robot, the use of the 
tablet and the implementation of the game on the tablet of the Pepper robot, the verbal answers 
from the Pepper robot, and the feedback in the game. 
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2 What did you not like about the “Make of Do” Game with the Pepper robot? Students criticised 
the following elements which they did not like about the game: missing display about system state, 
too few gestures of the robot before and after the game, the very loud volume of the voice of the 
robot in the English language, and the fact that the Pepper robot did not read the sample 
sentences aloud. 


3 Which recommendation do you have for the improvement of the “Make of Do” Game with the 
Pepper robot in the next iterations? The following recommendations on improving the human- 
robot interaction were expressed by the participants: include a system display, explain rules of 
the game briefly and concisely before beginning the game, incorporate gestures of the robot more 
explicitly into the gameplay, adjust the volume, increase eye contact. The participants also 
formulated a few recommendations related to the content of the game and recommended using 
other tense forms for make and do phrases, and to end sample sentences with a selection of 
suitable signal words (such as always, yesterday, today). 


4 CONCLUSIONS 


This paper presented the design and programming of the application “Make or Do” Game with the 
humanoid robot Pepper and the results from a pilot study with a small group of five university students, 
who played the game with Pepper and evaluated the human-robot interaction. The results from the pilot 
study indicate that the expectations of the students towards a robot-led language learning game were 
met to a large extent and that the participants perceived the Pepper robot as a likeable, friendly, kind, 
pleasant, and nice game partner, who was merry, funny, in a good mood, relaxed, and interested. 
Furthermore, both the Likeability Scale from the Godspeed questionnaires and the newly developed 
semantic differential scale for the assessment of the mood of the robot showed excellent internal 
consistency. Finally, the participants in the pilot study enjoyed interacting with the Pepper robot and 
considered using the tablet as a helpful element in the game. However, there were also a few critical 
points, primarily related to the high volume of robot’s voice and to an insufficient inclusion of gestures. 
In summary, it could be observed that the participating students experienced the “Make or Do” game 
with the Pepper robot as an enjoyable and exciting experience. However, following up on the research 
about the possible positive influence of the “novelty effect” on learners’ willingness to engage with 
humanoid robots, which for most learners are new and previously unencountered classroom 
technologies [8], [9], [10], the authors of this paper recommend conducting further research based on 
longer term studies and using larger sample sizes to properly assess the pedagogical effectiveness of 
game-based RALL using humanoid robots. 
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